Topic Shift Analysis & Validation

Author

Doron Feingold

Published

September 7, 2025

Overview

Now that we have labels for out topic we can plot the topic shift and review statistics that might help us understand trends across the different models.

These metrics describe the behavior and nature of each topic over time.

Topic Volatility

How “spiky” or “stable” is a topic’s prominence over time?

For each topic, calculate the standard deviation or coefficient of variation of its average gamma scores across all speeches.

Interpretation: A high volatility score indicates an “event-driven” topic that appears intensely and then fades (e.g., war, pandemic). A low score suggests a stable, persistent “background” topic that is always present.

Topic Persistence

Does a topic remain relevant long after its peak, or does it disappear completely? For each topic, calculate a “persistence score” by dividing its average gamma in the most recent year by its all-time peak gamma.

Interpretation: A score close to 1 means the topic is still highly relevant today, even if its peak was long ago. A score close to 0 means the topic has largely faded from the discourse.

Topic Exclusivity

How unique are the words that define a topic? For each topic, count how many of its top 15 words are not in the top 15 words of any other topic in the same model.

Interpretation: High exclusivity suggests a very distinct and well-defined theme. Low exclusivity suggests a more general or foundational topic whose vocabulary is shared across many different policy areas.

4 Topics

We use the 4 topic model for more defined and interpretable topics.
Topic Statistics and Cadence (Yearly)
Topic Peak Year Volatility Persistence Exclusivity
International Affairs & Defense 1878 0.45 0 17
Parliamentary Procedure & Administration 1946 0.37 0 12
Economic Development & Growth 1980 0.32 0 9
Social Policy & Community Welfare 2013 0.34 1 16

The plot is striking. The topics are well defined and a clear pattern of a synchronized raising and declining prominence (stable Volatility). The topic labels seem to fit well with the eras each topic represents.

8 Topics

The 8 topics model is more temporal. Even though there more topics they are evenly exclusive.

Topic Statistics and Cadence (Yearly)
Topic Peak Year Volatility Persistence Exclusivity
Federal-Provincial Coordination 1910 0.45 0.00 12
Parliamentary Business & Trade 1931 0.30 0.00 9
War & International Relations 1944 0.28 0.00 10
Jobs & Economic Policy 1968 0.20 0.00 5
Infrastructure & Regional Development 1976 0.21 0.00 8
Economic Development Programs 1983 0.23 0.40 6
Indigenous Affairs & Climate 2013 0.23 0.40 7
Social Programs & Healthcare 2020 0.24 0.21 8

We now see topic that declined in prominence and later reemerge, like “War & International Relations”. We see periods where topics peek with less prominence. Topics in the early and late years, appear to have strong prominence.

Validation against Major Events

To validate the topics prominence we compare them against major events and governments.

Major Canadian Historical Events

The table below lists partial but uncontroversial, major Canadian historical events marked in the plots below for.

Major Canadian Historical Events Since 1867
Date Event
1867 Confederation: The Dominion of Canada is formed.
1885 Completion of the Canadian Pacific Railway.
1914 Canada enters World War I.
1918 World War I ends.
1929 Great Depression begins.
1939 Canada enters World War II.
1945 World War II ends.
1959 Suez Crisis
1973 Oil Embargo
1982 Canada Act 1982: Patriation of the Constitution.
1995 Widespread adoption of the internet.
2001 Canada joins the Afghanistan War.
2020 COVID-19 pandemic: WHO declares a global pandemic.

Governments

We can also compare the topics prominence against the different governments and see if they “fit”.

Conclusion

This topic modeling analysis of Canadian Speeches from the Throne reveals meaningful patterns in political discourse that align closely with historical events and governmental transitions. The 4-topic model demonstrates clear temporal succession, with Parliamentary Procedure & Administration dominating the early confederation period, followed by International Affairs & Defense during the world wars, Economic Development & Growth in the post-war boom, and Social Policy & Community Welfare in the modern era. The 8-topic model provides additional granularity, capturing the cyclical nature of themes like “War & International Relations” that resurge during different conflicts, and revealing how contemporary issues like Indigenous Affairs & Climate have gained prominence in recent decades. The validation against major historical events—from Confederation through both World Wars, the Great Depression, and the COVID-19 pandemic—confirms that these computational methods successfully identify genuine shifts in political priorities.

These findings demonstrate that unsupervised machine learning can effectively uncover the evolution of political discourse, providing quantitative evidence for how Canadian political priorities have shifted across nearly 160 years of parliamentary debate. The use of Speeches from the Throne as our corpus was particularly valuable, as these ceremonial addresses represent a standardized format delivered at consistent intervals, creating a reliable yardstick for measuring thematic changes over time. Unlike other parliamentary speeches that vary widely in context and purpose, Throne Speeches serve as an index of governmental priorities, allowing us to track shifts in political focus with greater precision and comparability across different eras and administrations.